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Introduction 


Lack  of  acquisition  guidance  is  a  major  concern  for  projects  involved  in  the 
acquisition  and  sustainment  of  systems,  including  software-intensive  systems. 
Over  the  past  decade,  much  of  the  headquarters  and  field-level  acquisition 
guidance  for  systems  and  software  acquisition  and  sustainment  has  been 
rescinded,  simplified,  or  reduced  in  scope  such  that  only  minimal  acquisition- 
related  guidance  remains  in  many  acquisition  areas. 

This  reduction  of  guidance  has  occurred  as  system  complexity  and  the 
software  contribution  to  overall  system  functionality  rises  to  unprecedented 
levels. 

Congressional-  and  DOD-level  guidance  continues  to  emphasize  software 
acquisition  process  improvement,  including  the  measurement  of  process 
performance  by  acquisition  organizations 

The  goal  of  this  tutorial  is  to  define  effective  and  efficient  acquisition  practices, 
both  directed  internally  toward  the  acquisition  project  and  directed  externally 
toward  the  monitoring  and  control  of  the  selected  supplier(s).  These  practices 
are  intended  to  provide  a  basis  for  acquisition  process  discipline  while 
balancing  the  need  for  agility. 
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Problem  Statement 

Many  DoD  contractors  claim  high  Maturity  Levels  (3  and 
above)  as  measured  by  the  Capability  Maturity  Model 
Integration,  yet  from  the  perspective  of  acquisition  program 
managers  on  some  high  visibility  individual  programs,  for 
various  reasons,  individual  teams  are  not  executing  to  the 
level  claimed  in  proposals. 
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Example  Program 

Background 

Large  DoD  program  with  multiple,  geographically  dispersed 
engineering  locations. 

Multi-contractor  teams  (10+)  using  different  processes. 

Several  million  lines  of  code. 

Systems  engineering  challenges. 

Combination  of  legacy,  re-use,  COTS  integration  and  new 
development. 

All  contractor  sites  are  Maturity  Level  3  or  higher. 

18  months  after  contract  award,  the  program  office  conducted  a 
CMMI  “Class  B”  appraisal  on  the  team. 
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Appraisal  Output 


30  n 


Project  Mgmt  Engineering  Process  Support 

Processes  Mgmt  Processes 


□  Number  of  Strengths  ■  Number  of  Weaknesses 


Project  Mgmt  Processes 

-  Project  Planning 

-  Project  Monitoring  &  Control 

-  Integrated  Project  Mgmt 

-  Risk  Management 

Engineering  Processes 

-  Requirements  Mgmt 

-  Requirements  Definition 

-  Technical  Solution 

-  Product  Integration 

-  Verification  (Peer  Reviews) 

Process  Mgmt 

-  Organizational  Process  Focus 

-  Organizational  Process  Definition 

Support  Processes 

-  Measurement  &  Analysis 

-  Product  and  Process  Quality 
Assurance 

-  Configuration  Mgmt 

-  Decision  Analysis 
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Example  Program 

Issues  Identified 

PROJECT  MANAGEMENT 

•  Lack  of  project  plans  or  having  only 
incomplete,  conflicting  or  out  of  date 
project  plans 

•  Ineffective  use  of  Integrated  Master 
Schedule  as  basis  for 
planning/tracking  status  across 
program 

•  Undefined  engineering  and 
management  processes  on  program 

•  Inability  to  track  and  manage  actions 
to  closure 

•  Inadequate  cost  estimation  processes, 
methods,  data  and  tools 

•  Inadequate  staffing  and  training 
project  personnel 

•  Tracking  dependencies  between  or 
across  teams  not  defined 

•  Managing  project  data  ad  hoc 

•  Inability  to  proactively  identify  and 
manage  risks 

©  2005  by  Carnegie  Mellon  University 
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ENGINEERING 

•  Lack  of  understanding  of  the 
program’s  requirements 

•  Inability  to  trace  requirements  to 
architecture/design  or  to  test 
plans/procedures 

•  Poor  linkage  of  functional  and 
performance  requirements 

•  Inconsistent  requirements 
management  at  different  levels 

•  No  criteria  for  making 
architectural/design  decisions  among 
alternatives 

•  Not  capturing  entire  technical  data 
package  (requirements,  design  and 
design  rationale,  test  results,  etc) 

•  Efficiency  of  design  process/methods 
in  question 

•  Late  definition  of  integration  and  test 
procedures 
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Example  Program 

Issues  Identified  2 

SUPPORT 

Difficult  to  identify  items  in  configuration  management  baselines 

Lack  of  ability  to  manage  individual  “versions”  in  incremental  development 

Inability  to  effectively  managing  changes  to  work  products  throughout  lifecycle 

Not  conducting  audits  to  establish/ensure  integrity  of  baselines  throughout 
incremental  engineering  and  development 

Inefficient  change  management  process  (cycle  time,  volume  of  changes) 
Roles/responsibilities  of  change  control  boards  not  defined 
Quality  Assurance  audits  of  products  and  processes  not  consistent 
QA  involvement  in  system  and  software  engineering  processes  not  consistent 
No  metrics  to  manage  engineering  activities  (outside  of  cost/schedule  data) 
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Example  Program 

Results 

Early  and  periodic  Class  B  appraisals  using  CMMI  identified  risks 
to  program  success 

Identified  risks  were  assigned  to  contractor,  to  acquirer,  or  both 
based  upon  who  was  best  able  to  mitigate  them. 

Many  risks  were  managed  jointly  and  cooperatively  between  the 
contractor  and  the  acquirer 

Identification  of  and  attention  to  risks  early  in  the  program  life 
cycle  led  to  the  ultimate  success  of  the  program. 
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High  Maturity 
Organizations 


WHY? 

Maturity  Levels  are  good 
indicators  of  organizational 
potential  performance. 

They  describe  how  the  next 
project  will  most  likely  perform 
based  on  a  sampling  of 
existing  projects. 

Maturity  Levels  reside  at  the 
organizational  level  and  are 
not  an  indication  of  how  an 
individual  project  is  performing. 
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The  Acquirer’s  Concern 


During 

source 

selection: 


Ongoing: 


•  How  capable  is  a 
contractor  team  to 
deliver  an 
operational 
capability? 

•  How  well  is  my 
program 
performing? 


Maturity  Levels  at  the  organizational  level  are 
necessary  but  not  sufficient  to  provide  answers  to 
these  questions  at  the  program  level. 
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Key  Questions 

Is  the  appraisal  of  the  contractor’s  organizational 

maturity  relevant  to  my  project? 

•  Did  the  part  of  the  organization  executing  my  project 
participate  in  the  appraisal? 

•  Did  projects  similar  to  mine  participate  in  the  appraisal? 

•  Are  the  appraised  processes  routinely  used  by  the  part 
of  the  organization  executing  my  project? 

•  Are  the  appraised  processes  an  integral  part  of  the 
project  execution,  or  are  they  an  overlay  on  the  “the  real 
way  the  work  gets  done”? 

The  BIG  question:  What  processes  will  really 

be  used  on  MY  project 
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Excerpt  from  Defense  Acquisition 
Guidebook  -  Source  Selection 

4.2.5.2  Capability  Reviews 

Capability  reviews  . . .  are  a  useful  tool 
available  during  source  selections  to  assess 
the  offerors'  capability  in  selected  critical 
process  areas.  Capability  reviews  may  be  the 
appropriate  means  for  evaluating  program- 
specific  critical  processes  such  as  systems 
engineering,  software  development, 
configuration  management,  etc.  ... 
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Getting  the  processes  that  you  need 

1 .  Identify  the  characteristics  of  the  processes  that  you  need  on 
your  project. 

2.  Include  process  evaluation  as  one  of  the  source  selection 
criteria. 

3.  Require  the  bidders  to  define  in  the  proposal,  the  processes 
and  the  process  outputs  that  they  intend  to  use  on  the  project. 

4.  During  source  selection,  evaluate  the  bidder’s  processes  w.r.t. 
the  project’s  process  needs  using  the  SCAMPI  method. 

5.  For  the  winning  bidder,  reference  the  proposed  processes  in 
the  contract. 

6.  During  contract  execution,  evaluate  the  contractor’ 
compliance  with  the  proposed  processes. 
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Step  1 :  Identify  Process  Needs  i 

Self-assessment  of  project  needs  by  the  PMO 

•  Based  upon  “Guidelines  for  using  CMMI  in  Acquisition 
Programs”  (draft) 


CMMI  Risk  Worksheet  Results 


Not 

Important 

Important 

Essential 

Project  Planning 

1 

2 

3 

4  5 

6  7 

8 

9 

10 

1  .Establish  Estimates 

0 

0 

0 

0  0 

0  • 

0 

0 

0 

2.Develop  a  Project  Plan 

0 

0 

0 

O  0 

0  0 

• 

0 

0 

3. Obtain  Commitment  to  the 

0 

0 

0 

0  0 

•  0 

0 

0 

0 

Plan 


Project  Monitoring  and  Control 
4. Monitor  Project  Against 
Plan 
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Step  1 :  Identify  Process  Needs  2 

Example 

•  Program  is  looking  to  select  a  Lead  System  Integrator  for  a 
complex,  multi-year,  development  effort 

•  Program  is  concerned  with  the  potential  LSI’s  ability  to: 

-  manage  risk 

-  manage  suppliers 

-  plan  and  track  the  program 

-  build  an  integrated  team 

-  develop  an  architecture 

-  integrate  the  various  components 

•  The  program  might  want  to  explore  the  bidders  proposed 
processes  in  these  CMMI  process  areas:  PP,  PMC,  IPM  (with 
IPPD),  ISM,  RD,  IT,  RSKM,  PI 
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Step  2:  Source  Selection  Criteria 

Don’t  just  require  organizational  Maturity  Levels 

Include  an  evaluation  of  contractor-proposed 
processes  as  one  of  the  factors  for  source  selection. 
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Step  3:  Require  process  proposals  i 

RFP  requires  all  bidders  to  propose  the  processes  that 
they  will  use  on  this  project. 

Process  proposals  include 

•  a  description  of  the  process 

•  a  list  of  the  outputs  /  artifacts  produced  by  the  process. 


©  2005  by  Carnegie  Mellon  University 


Tutorial:  Use  of  CMMI®  in  Acquisition  Environments  -  Page  19 


Carnegie  Mel  Ion 

Software  Engineering  Institute 

Step  3:  Require  process  proposals  2 

RFP  Language 

L.X.X.X  Product  Development  Capabilities 

In  support  of  the  Management  Factor  evaluation,  the  Government 
intends  to  conduct  an  evaluation  of  the  product  development  and 
management  capabilities  of  the  offeror  team  proposed  for  application  on 
this  project.  The  evaluation  will  involve  methods  and  procedures  tailored 
from  the  Software  Engineering  Institute  (SEI)  Standard  CMMISM 
Appraisal  Method  for  Process  Improvement  (SCAMPISM)  Class  X* 
Version  1.1  using  Capability  Maturity  Model®  Integration  (CMMISM)  for 
Systems  Engineering,  Software  Engineering,  and  Integrated  Product  and 
Process  Development  (CMMI-SE/SW/IPPD),  Version  1.1.  The  offeror 
shall  provide  the  SCAMPI  documentation  described  in  Attachment  A  of 
this  RFP  to  support  this  evaluation.  This  documentation  shall  be  provided 
in  the  proposal  and  will  not  be  included  in  the  page  count  limitations  for 
the  proposal. 


*B  or  C  depending  on  program  needs 
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Step  4:  Evaluate  process  proposals  i 

Appraise  the  proposed  processes  using  a  SCAMPI-C 
method. 

•  SCAMPI-C  is  suitable  for  process  appraisal  based  on 
document  review. 

•  Use  an  authorized  Lead  Appraiser  with  acquisition 
experience 

•  Ensure  the  appraisal  team  is  trained  and  experienced 

Perform  a  gap  analysis  against  the  self-assessment  of 
project  needs  established  in  Step  1. 

•  Gaps  represent  process-related  risks  to  the  project. 
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Step  4:  Evaluate  process  proposals  2 


If  the  program  is  of  sufficient  size,  duration,  or 
complexity,  consider  performing  SCAMPI-Bs  on  the 
bidding  “teams”. 

Use  the  continuous  representation,  select  3  to  7 
process  areas  based  on  program  risk. 

Use  an  authorized  Lead  Appraiser  with  acquisition 
experience. 

Ensure  the  team  is  trained  and  experienced. 
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Step  5:  Contract  proposed  processes 

Reference  the  proposed  processes  in  the  awarded 
contract. 

•  Don’t  tell  the  contractor  what  processes  to  use. 

-  That  could  shift  performance  liability  to  the  PMO  if  the 
processes  are  found  to  be  unsuitable  for  the  project. 

•  Just  tell  the  contractor  to  perform  as  proposed. 
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Step  6:  Evaluate  process  compliance  i 


Monitor  the  production  of 
process-related  artifacts 

Perform  one  or  more 
SCAMPI-B  appraisals  on 
key  process  areas  to 
assess  compliance  to 
proposed  processes 

•  Appraisal  focus  is  on 
contract  monitoring,  not 
process  improvement 
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Excerpt  from  Defense  Acquisition 
Guidebook  -  Contract  Monitoring 

4.2.5.3  Capability  Appraisals 

...the  program  manager  retains  the 
right ...  to  independently  evaluate  the 
process  capabilities  of  the  selected 
team  prior  to  or  immediately  after 
contract  award.  ...  Periodic 
appraisals  are  encouraged  as  part  of 
contract  process  monitoring 


activities.  ...  assessments  are  most 
valuable  when  they  apply  across  the 
full  program  team,  and  not  just  one 


Tutorial:  Use  of  CMMI®  in  Acquisition  Environments  -  Page  24 


Carnegie  Mel  Ion 

Software  Engineering  Institute 

Step  6:  Evaluate  process  compliance  2 

Contract  Monitoring  Guidelines 

•  Don’t  appraise  Maturity  Levels 

•  Use  the  SCAMPI-B  method 

•  Include  the  entire  program  team  (prime,  subs,  and  the 
acquisition  program  office) 

•  Use  the  continuous  representation,  select  process  areas 
based  on  program  risk  (selection  may  evolve  during  program 
life). 

•  Use  an  authorized  Lead  Appraiser 

•  Ensure  the  team  is  trained  and  experienced 

•  Include  contractor  members  on  the  appraisal  team 

•  Use  the  results  as  the  basis  for  collaborative  risk  mitigation 
and  process  improvement  across  program  team 
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Benefits  of  Using  the  SCAMPI  Family 
of  Appraisal  Methods 

SCAMPI  Class  B  and  Class  C  appraisals  are  consistent  with  the 
SCAMPI  Class  A  Method  (same  steps,  use  of  Plls,  etc.)  and: 

•  are  led  by  an  authorized  team  lead 

•  require  team  training 

•  focus  on  areas  of  risk  to  the  program  without  an  artificial  focus 
on  achieving  “Levels” 

•  are  repeatable 

•  follow  a  publicly  vetted,  documented,  and  easily  accessible 
method 

In  the  absence  of  a  repeatable  method  with  clear  expectations  on 
the  part  of  the  acquirer  and  contractor  communities,  “home 
grown”  methods  will  emerge. 
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Summary 

Maturity  Levels  alone  do  not  provide  the  information  an 
acquirer  needs  to  determine: 

•  How  capable  is  a  contractor  team  to  deliver  an 
operational  capability? 

•  How  well  is  my  program  performing? 

Acquirers  need  a  simple,  actionable  set  of  guidelines  on 
how  to  use  the  CMMI  framework  (models  and  appraisal 
methods)  to  help  reduce  program  risk. 
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The  State  of  Acquisition  Practices 


Capability  Maturity  Model  Integration 
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What  is  “Acquisition” 
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The  State  of  Acquisition  Practice  ^ 

The  agencies  assume  the  partnership  arrangement 
absolves  them  of  all  acquisition  management 
responsibilities...” 

Virtually  all  (Air  Force)  software-intensive  systems  suffer 
from  difficulties  achieving  cost,  schedule,  and 
performance  objectives. 

“I'd  rather  have  it  wrong  than  have  it  late.”  A  senior  manager  (industry) 

“The  bottom  line  is  schedule.  My  promotions  and  raises 
are  based  on  meeting  schedule  first  and  foremost.”  A  program 

manager  (government) 

Lack  of  robust  systems  engineering  practices  identified 
as  critical  factor  in  SBIRS-High  problems.  Lt.  Gen.  Brian  A.  Arnold, 

USAF,  CDR,  USAF/SMC 
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The  State  of  Acquisition  Practice  2 

Is  There  an  Acquisition  Crisis? 

Investigation  of  one  acquisition  program  showed: 

•  System  complexity  and  the  program’s  lack  of  experience  in 
procuring  major  systems  caused  serious  cost  growth. 

•  Program  lacks  systems  engineering  and  program 
management  expertise. 

•  Absence  of  requirements  stabilization  process. 

•  Program  management  does  not  enforce  timely  milestones, 
timelines,  and  deliverables. 

•  Program’s  lack  of  process  control  made  assessment  of 
technical  risk  impossible. 

•  Program’s  lack  of  short-  and  long-term  budget  tracking  makes 
cost  assessment  nearly  impossible. 

•  Program  does  not  manage  risk. 
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The  State  of  Acquisition  Practice  3 

What’s  the  Problem? 

There  are  many.  Among  them, 

*  Evidence  shows  that  an  acquirers  management 
processes  and  practices  and  resultant  decisions  can 
have  a  negative  impact  on  the  development  processes 
of  the  supplier 

•  A  mismatch  in  Acquirer/Supplier  in  terms  of  associated 
process  capability  and  maturity  can  have  unpredictable 
and  even  disastrous  results. 


And  the  challenges  are  increasing  ... 
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Complexity  in  Modern  Systems 

Many  commercial  products  are  the  result  of  a  complex  mix  of 
subcomponents  engineered  into  a  system 

Most  DoD  weapon  and  information  systems  are  at  least  this 
complex 


Aclive  Roll  Stabilization  \AF\3)  Optional  rear- seat  ai  de-i  mpacl 
From -seal  Head  O  airbags  (deacli  vabon  option  > 
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V-fi  en  gl  ne  Restrai  n  ta  tor  Iron  t  seats 

on  low  beam  a  d  20-way  powar  O  Optional  Sport  Package 

front  comlort  aeats  ^ 


high -beam  headlights  with 
dynamic  aulo-lavellng  0 

Massive  AE35  ventilated 
di  so  brakes  with 
Dynamic  Brake  Control 


Dynamic  Stability  Control  i  DSC) 
with  All  Season  Traction  >|ASC)  and 
Dynamic  Traction  Control  (DTCt 
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Weapon  System  Complexity 
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System  of  Systems  Complexity 
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Increasing  System  Complexity 
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Capability  Maturity  Model  Integration 
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What  Can  Be  Done? 

Based  on  the  premise  that 

The  quality  of  the  product  is  governed  largely 
by  the  process  used  to  create  the  product 

We  could  improve  the  Supplier’s  process  and  practices 

•  But  the  developers  have  a  head  start  (CMMI-based  improvement 
programs  are  widespread) 


We  could  improve  the  Acquirer’s  processes  and 
practices  by: 

•  increasing  the  visibility  of  the  acquirers  contribution  to  program 
success 

•  defining,  implementing,  measuring  and  evolving  effective 
acquisition  processes  and  practices 
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How  Do  You  Want  to  Work? 


•  Random  motion  -  lots  of 
energy,  not  much  progress 

•  No  teamwork  -  each  person 
goes  his  own  way 

•  Frequent  conflict 

•  You  never  know  where  you’ll 
end  up 


•  Directed  motion  -  every  step 
brings  you  closer  to  the  goal 

•  Coordinated  efforts 

•  Cooperation 

•  Predictable  results 


Process  can  make  the  difference 
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Focus  of  CMMI 


♦ 


♦ 


SW-CMM  is  applied  here 
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CMMI  -  Continuous  SE/SW/IPPD/SS 

CMMI 


I 

_ _ 

l 

Process 

Management 

Project 

Management 

Engineering 

Support 

•  Organizational  Process 

•  Project  Planning 

•  Requirements 

•  Configuration 

Focus 

•  Project  Monitoring  and 

Management 

Mgmt. 

•  Organizational  Process 

Control 

•  Requirements 

•  Process  and 

Definition 

•  Supplier  Agreement  Mgmt. 

Development 

Product  Quality 

•  Organizational  Training 

•  Integrated  Project  Mgmt. 

•  Technical  Solution 

Assurance 

•  Organizational  Process 

•  Risk  Management 

•  Product  Integration 

•  Measurement  & 

Performance 

•  Integrated  Teaming 

•  Verification 

Analysis 

•  Organizational 

Innovation  and 
Deployment 

•  Integrated  Supplier  Mgmt 

•  Quantitative  Project  Mgmt. 

•  Validation 

•  Decision  Analysis 
and  Resolution 

•  Organizational 
Environment  for 
Integration 

•  Causal  Analysis 
and  Resolution 
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Structure  of  CMM1 1 
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AREA  2 


Generic 

Goals 
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For  CMMI-SW/SE 


22  Process 
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157  Goals 


539  Practices 
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Perspectives  on  Maturity 
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CONTRACTOR  AND  PROCESS 

What  Levels  Tell  Us 

Levels  are  good  indicators  of  potential 
organizational  performance 

They  describe  how  the  next  project  could  perform 
based  on  a  sampling  of  existing  projects 

Capability  Levels  and  Maturity  Levels  reside  at  the 
organizational  level  (corporation,  major  division) 
and  are  not  an  indication  of  how  any  individual 
project  is  performing 


Note:  Sometimes  a  project  is  large  enough  to  be  considered  an  organizational  unit  (e.g.  JSF,  C-17) 
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Summary 

Acquisition  is  a  challenging  multi-disciplinary  effort 
occurring  in  a  difficult  environment,  and  demands  for 
greater  capabilities  and  increasing  complexity  are  adding  to 
this  challenge. 

Capable  performance  by  BOTH  the  acquirer  and  the 
supplier  are  essential  to  program  success 

A  focus  on  PROCESS  at  the  acquirer  and  at  the  supplier 
can  help. 

CMMI  is  a  proven  and  widely  accepted  process 
improvement  model 
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Module  2  Agenda 

Introduction  to  the  CMMI  Acquisition  Module 

Project  Management  Process  Areas 

•  Project  Planning 

•  Project  Monitoring  and  Control 

•  Solicitation  and  Contract  Monitoring 

•  Integrated  Project  Management 

•  Risk  Management 
Summary 
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Where  Does  Process  Fit  in 
Acquisition? 

...at  the  Project  Management  Office  (PMO) 

•  Management  of  internal  PMO  activities 

•  Management  of  processes  applied  to  project 

•  Oversight  of  contractors’  processes 

•  Integration  of  contractors’  and  PMO  processes 

...  at  the  Contractor 

•  Management  of  internal  contractor  activities 

•  Oversight  of  subcontractor  processes 

...  for  integration  of  PMO,  contractor,  and 
subcontractor  processes 


©  2005  by  Carnegie  Mellon  University 


Tutorial:  Use  of  CMMI®  in  Acquisition  Environments  -  Page  4 


Carnegie  Mel  Ion 

Software  Engineering  Institute 


PMO  AND  PROCESS 

Process  and  the  Roles  of  the  PM 


Manage  process  within  the  PMO 


Manage  process  applied  to  the  project 


Suppliers 


Exercise  oversight  of  the  contractors 
process  management 


Ensure  integration 
of  contractor  and 
PMO  processes 
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PMO  AND  PROCESS 

The  PMO  Management  Role 


The  PM  is  responsible  for  managing  internal  PMO 
processes.  The  PM  must  take  a  hands-on  approach  to 


•  Identify,  define,  and  document  process  needs 


•  Communicate  and  train  the  PMO  staff 


•  Support,  track,  measure, 
and  review  the  PMO 
processes 
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PMO  AND  PROCESS 

Program  Management  Role 


Define  the  interface  between  the  PMO  and  the 
contractor  using  the  RFP  and  negotiations 

•  Project  process  requirements 

•  Project  metrics 

•  Project  communication  needs 

•  Project  risk  management  needs 


Manage  the  interface  during 
contract  execution 

•  Real-time  monitoring  of 
deliverables 

•  Keep  communication 
channels  clear  &  open 

•  Develop  trust  with  contractor 


PMO 


?rojeQ, 
Contract 
Communication  | 
Program  Risk 


.hi 


Contractor 
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PMO  AND  PROCESS 

Contractor  Oversight  Role  i 

Process  maturity  of  the  contractor  should  be  a 

consideration  in  source  selection 

•  Obtain  process  definitions  and  commitments 

-  Just  requiring  a  CMMI  Maturity  Level  is  NOT  enough. 

-  You  need  to  ensure  that  high-maturity  processes  are  applied 
to  YOUR  project 

-  Require  your  bidders  to  define  the  processes  they  will  use  in 
their  proposals 

-  Evaluate  the  proposed 
processes  as  a  part  of 
source  selection 

-  Reference  the  processes 
in  the  contract 

•  Plan  process  integration 


Hint 
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PMO  AND  PROCESS 


Contractor  Oversight  Role  2 


After  contract  award,  ensure  that  contracted  process 
commitments  are  kept 

•  Committed  processes  are  used  by  the  project  team 

•  Process  artifacts  are  evident 

•  Process  integration  is  effective  and  monitored 

•  Consider  periodic  independent  appraisals  of  key  process  areas 
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PMO  AND  PROCESS 

Subcontractor  Oversight  Role 


For  many  systems,  the  bulk  of  the  work  is  done  by 
subcontractors 


Primary  responsibility  for  oversight  of 
with  the  prime  contractor 

PMO  role  is  to  ensure  that  prime  is 
providing  adequate  oversight  to 
subcontractors 


•  Ensure  flowdown  of  project 
process  requirements 

•  Ensure  integration  of 
prime  and  subcontractor 
processes 


Hint 
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PMO  AND  PROCESS 


Process  Integration  Role 


It  is  the  PMO’s  responsibility  to  ensure  PMO  and 
Contractor  processes  are  compatible 

•  Include  any  process  “must  haves”  in  the  RFP 

-  Consider  specific  compatibility  with  tools  for  risk, 
requirements,  schedule,  etc. 

•  Ensure  good  communications  with  contractor(s)  regarding 
process  incompatibilities 

•  Integration  focus 


needed  throughout 
project 
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CMMI  Acquisition  Module  (CMMI-AM) 

Focuses  on  effective  acquisition  activities  and  practices  that 
are  implemented  by  first-level  acquisition  projects  (e.g., 
System  Project  Office/Program  Manager) 

Acquisition  practices  drawn  and  summarized  from  existing 
sources  of  best  practices: 

-  Software  Acquisition  Capability  Maturity  Model  (SA-CMM) 

-  Capability  Maturity  Model  Integration  (CMMI) 

-  FAA  Integrated  Capability  Maturity  Model  (iCMM) 

-  Section  804 

Intended  to  be  used  in  conjunction  with  the  CMMI  as  an 
acquisition  “lens”  for  interpreting  the  CMMI  in  acquisition 
environments 


CMMI-AM  -  a  tool  for  the  acquirer 
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CMMI-AM  Structure 


Project  Planning 
Project  Monitor  and 
Control 

Integrated  Project 
Management 
Risk  Management 

Solicitation  and 
Contract  Monitoring 


Requirements  Management 
Requirements  Development 
Verification 
Validation 


Measurement  and 
Analysis 

Decision  Analysis  and 
Resolution 

Transition  to  Operations 
and  Support 


Key 

H  New  for  CMMI-AM 
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Structure  of  CMMI-AM 


i 


For  CMMI-SW/SE 

12 

Process 

Areas 

28 

Goals 

246 

Practices 

||  ...  plus  47  self-assessment  questions  ] 
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Module  2  Agenda 

Introduction  to  the  CMMI  Acquisition  Module 

Project  Management  Process  Areas 

•  Project  Planning 

•  Project  Monitoring  and  Control 

•  Solicitation  and  Contract  Monitoring 

•  Integrated  Project  Management 

•  Risk  Management 
Summary 


©  2005  by  Carnegie  Mellon  University 


Tutorial:  Use  of  CMMI®  in  Acquisition  Environments  -  Page  15 


Carnegie  Mel  Ion 

Software  Engineering  Institute 

Project  Management  PAs 


Project  management  process  areas  cover  the  project 
management  activities  related  to  planning,  monitoring,  and 


controlling  the  project. 

Project  Planning  PP 

Project  Monitoring  and  Control  PMC 

Solicitation  and  Contract  Monitoring  SCM 
Integrated  Project  Management  IPM 

Risk  Management  RSKM 
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Project  Planning 

The  purpose  of  project  planning  is  to  establish  and 
maintain  plans  that  define  project  activities.  For 
acquisition: 

•  Project  planning  starts  by  setting  the  acquisition  strategy  and 
is  followed  by  planning  the  acquisition  process  in  ever 
increasing  levels  of  detail 

•  As  the  acquisition  proceeds  toward  selection  of  a  supplier, 
the  supplier’s  planning  process  should  be  reviewed  for 
sufficiency 

•  The  resulting  plans  should  also  be  reviewed  for  consistency 
with  the  system  acquisition  plans 

•  The  acquirer’s  and  developer’s  project  planning  processes 
are  continuous  and  the  plans  evolve  to  meet  the  project’s 
needs. 
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Purpose  of  Acquisition  Planning 

Guide  program  execution 

•  From  initiation  through  re-procurement  and  during  post¬ 
production  support 

•  Systems,  subsystems,  components,  spares,  and  services 

Minimize  the  time  and  cost  of  satisfying  identified,  validated 
needs  in  a  manner  consistent  with  common  sense  and 
sound  business  practices 

Planning  evolves  through  an  iterative  process  and  becomes 
increasingly  more  definitive  in  describing  the  relationship  of 
the  essential  elements  of  a  program 

Paraphrased  from  DoD  5000  Interim  Guidebook 
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Poor  Project  Planning  ... 

Symptoms 

•  Poor  estimates  lead  to  cost  and  schedule  overruns. 

•  An  inability  to  discover  deviations  from  undocumented  plans. 

•  Resources  are  not  available/applied  when  needed. 

•  An  inability  to  meet  commitments. 

•  Project  failure. 

Why  should  we  care? 

•  Customers  don’t  trust  acquirers  or  suppliers  who  waste  their 
resources  (i.e.,  loss  of  future  business). 

•  No  lessons  learned  for  future  projects  means  making  the  same 
mistakes  on  multiple  projects. 

•  Unhappy  customers,  employees,  and  stakeholders  means  a 
short  life  for  the  business. 

“If  you  fail  to  plan,  then  you  plan  to  fail.” 
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Acquisition  Strategy  vs.  Acquisition  Plan 

Acquisition  Strategy  is  high-level 

•  “Top-level  road  map  for  program  execution  from  program 
initiation  through  post-production  support.” 

•  ITERATIVE  -  should  be  updated 

•  Level  of  detail  changes  as  you  go  through  the  phases 

•  As  per  DoDI  5000.2  required  for  ALL  programs  at: 

-  Program  Initiation  for  Ships 

-  Milestone  B 

-  Milestone  C 

-  Full-Rate  Production  Deployment  Review 

Acquisition  Plan  is  typically  for  one  phase 

Required  by  the  Federal  Acquisition  Regulation  (FAR) 
Focuses  on  specifics  of  the  acquisition 
Concerned  with  contract  type,  incentives,  etc. 
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Acquisition  Planning  Cycle 


x 


AoA 

Plan 


Tech 

Devel 

Strat 


J= 


Develop 

Acquisition  Strategy 


Acq. 

Strategy 

vl.O 


Evaluate 

Incremental 

Progress 


MS  B 


X 


Refine 

Acquisition  Strategy 


Acq. 

Strategy 

v2.0 


Refine 

Acquisition  Strategy 


Execute  Evaluate 

Acquisition  Incremental 
Strateav  Progress 


Execute 
Refined 
Acquisition 
Stratet 
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Acquisition  Planning  Objectives 

Communicate! 

•  Identify  risks 

-  Strategies  for  risk  mitigation 

-  Balance  risks  with  cost,  schedule  and  performance 

•  Define  expectations  for  all  stakeholders 

-  Role  and  responsibilities  of  all  parties 

•  Determine  how  to  make  your  program  executable  within 
budget  and  schedule  constraints 

-  Expected  program  changes  throughout  lifecycle 
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Acquisition  Strategy  Elements 

Acquisition  Approach 
Requirements 
Risk  Management 
Design  Considerations 
Business  Strategy 
Program  Management 
Support  Strategy 

From  Interim  Defense  Acquisition  Guidebook,  30  Oct  2002 
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Single-Step  and  Evolutionary  Acquisition 


100%  of  requirements 
known  at  start 


Single-Step 


‘Deliverable’  Capability 


100% 

★ 


100%  of  requirements 
known  at  start 


Incremental 


Known  increment  j  Known  increment  Known  increment 


Only  first  increment 
requirements  known  at  start 


Spiral 


Partially 


Known  increment  known 
. ►  increment  Unknown 


i 

j 

w  increment 

i 

User,  developer,  tester, 
sustainer  “use  and  learn” 

J 

j 

Unknown 

increment 


Time 


Based  on  AF  Program  Manager  Workshop  presented  by  Mr  Little 
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Evolutionary  Approach 


CR 

TD 

SDD 

PD 

OS 

Increm 

ent  1 

i _ 

0 

0 

Increment  2 


=  Contractor  Spiral 
Development 


B 


=  Contractor  Incremental 
Development 


Increment  3 


Adapted  from  dod5000.dau.mil 
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Acquisition  &  Development  Methods 


Single  Step  Acquisition,  Contractor  Incremental  Development 

Acquisition  of  a  New  Utility  truck 

Increment  1  -  Hard  to  produce  brakes 

★ 

increment  2  -  Easier  to  produce  brakes 

Evolutionary  Acquisition  (Spiral),  Contractor  Mixed  Development 
Inc  1  -  HW  Upgrades  A  -  Comms  System 

Single  Step  Development 

w 

Inc  2-SW  radios  for  existing  interfaces 

★ 

Increment  1-  Interface  1 

Increment  2-  Interface  2 

i 

Inc  3  -  Develop  new  interfaces 

Spiral  1  -  Prototype  1 

=  fielded  system 

Spiral  2  -  Prototype  T 

Spiral  3  -  Prototype  3 " yf 
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Program  Drivers 


What  software  and  system  issues  might  DRIVE 


your  acquisition  strategy 
to  successful  execution? 

•Schedule 

•Funding 

•Requirements  Stability 
•External  Interfaces 
•Deployment 

•Interoperability  (Programmatic 
and  Developmental) 

•Technology  Maturity 


due  to  the  risk  they  pose 

•Staffing 

•Test  Requirements 

•User  Support 
•Policy  Mandates 
•Security 

•System  Complexity 
Precedented  /  Unprecedented 
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Dealing  with  Drivers 

Determine  which  present  the  highest  risk  exposure  to  your 
program 

Determine  how  the  drivers  will  influence  your  acquisition 
strategy  elements 

•  Formulate  strategies  that  you  believe  will  deal  with  the 
risks  posed  by  the  top  drivers 

Analyze  the  strategies  to  determine  gaps  and  remaining 
high  risk  areas 
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Acquisition  Plan  Contents 

Acquisition  background  and  objectives 


•  Statement  of  need 
•Cost 

•  Risks 

•  Delivery  or  performance-period 
requirements 

Plan  of  action  (sample) 

•Sources 

•Source-selection  procedures 
•Budgeting  and  funding 

•  Make  or  buy 
•Test  and  evaluation 

•  Security  considerations 


Applicable  conditions 
Capability  or  performance 
Trade-offs 


Competition 

Acquisition  considerations 

Government-furnished  property 

Inherently  governmental  functions 

Logistics  considerations 

Contractor  versus  Government 
performance 
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Project  Planning 

CMMI-AM  Goals  and  Practices  ^ 

Specific  Goal  Specific  Practice 


Establish 

Estimates 

Estimate  the  Scope  of  the  Project 

Establish  Estimates  of  Work  Product  and  Task 
Attributes 

Define  Project  Life  Cycle 

Determine  Estimates  of  Effort  and  Cost 

Develop  a 
Project  Plan 

Establish  the  Budget  and  Schedule 

Identify  Project  Risks 

Plan  for  Data  Management 

Plan  for  Project  Resources 

Plan  for  Needed  Knowledge  and  Skills 

Plan  Stakeholder  Involvement 

Establish  the  Project  Plan 

Obtain 
Commitment 
to  the  Plan 

Review  Plans  that  Affect  the  Project 

Reconcile  Work  and  Resource  Levels 

Obtain  Plan  Commitment 
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Project  Planning 

Goal  1:  Establish  Estimates  i 

Estimates  of  project  planning  parameters  are 
established  and  maintained _ 

Establish  a  top-level  WBS1  to  estimate  the  scope  of  the 
project 

•  Defines  tasks  for  the  ENTIRE  project,  including  efforts  of: 

-  The  supplier  -  The  acquirer 

-  Other  stakeholders  (e.g.,  test  community,  users) 

•  Based  upon  product  architecture 

Establish  estimates  of  work  product  and  task  attributes 

•  Provides  a  basis  for  cost  and  effort  estimation 

•  Software  examples  -  KSLOC,  function  points,  #  of  objects,  #  of 
interfaces,  data  volume,  etc. 

1  See  MIL-HDBK-881 A  (  http://dcarc.pae.osd.mil/881handbook/881A.pdf ) 
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Project  Planning 

Goal  1:  Establish  Estimates  2 

Define  the  project  life-cycle  phases  upon  which  to  scope  the 
planning  effort 

•  Acquisition  method 

-  Single-Step  -  Evolutionary-incremental  Evolutionary-spiral 

•  Life  Cycle  phases 

-Development  -Manufacturing  -Verification  -Training 

-  Deployment  -  Operation  -  Support  -  Disposal 

Estimate  the  project  effort  and  cost  for  the  work  products 
and  tasks  based  on  estimation  rationale 

•  Define  estimation  rationale 

•  Estimate  cost  and  effort  for  each  work  product  and  task 

•  Consider  independent  review  of  estimates 
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Project  Planning 

Goal  2:  Develop  a  Project  Plan  i 

A  project  plan  is  established  and  maintained  as  the 
basis  for  managing  the  project 

Establish  and  maintain  the  project’s  budget  and  schedule 

•  Identify  assumptions,  constraints,  major  milestones 

•  Identify  task  dependencies 

Identify  and  analyze  project  risks 

•  Involve  stakeholders  in  identification  of  risk 

•  Analyze  impact,  timeframe,  and  probability  of  occurrence 

Plan  for  the  management  of  project  data 

•  Create  master  list  of  data  to  be  managed  (formal  and  informal) 

-  Identify  needs  for  version  control  and  configuration  mg’t 

•  Define  data  content  and  formats 

•  Establish  requirements  for  security  and  information  assurance 
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Project  Planning 

Goal  2:  Develop  a  Project  Plan  2 

Plan  for  necessary  resources  to  perform  the  project 

•  Identify  and  plan  for  process  requirements 

•  Identify  and  plan  for  staffing  requirements 

•  Identify  and  plan  for  facilities  and  equipment  requirements 

Plan  for  knowledge  and  skills  needed  to  perform  the 
project 

•  Identify  skills  needed 

•  Assess  available  skills 

•  Develop  a  plan  to  fill  the  gaps 
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Project  Planning 

Goal  2:  Develop  a  Project  Plan  3 

Plan  the  involvement  of  identified  stakeholders 

•  Identify  relevant  stakeholders 

•  Plan  their  involvement 

•  Obtain  commitments  for  involvement 

Establish  and  maintain  the  overall  project  plan  content 

•  Captures  all  relevant  planning  items  to  enable  communication 
among  the  project  team  and  stakeholders 

•  May  be  comprised  of  multiple  plans  such  as 

-  Integrated  Master  Plan 

-  Integrate  Master  Schedule 

-  Systems  Engineering  Management  Plan 

-  Software  Development  Plan 

•  Must  be  maintained  throughout  the  acquisition 
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Module  2  Agenda 

Introduction  to  the  CMMI  Acquisition  Module 

Project  Management  Process  Areas 

•  Project  Planning 

•  Project  Monitoring  and  Control 

•  Solicitation  and  Contract  Monitoring 

•  Integrated  Project  Management 

•  Risk  Management 
Summary 
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Project  Monitoring  and  Control., 

The  purpose  of  project  monitoring  and  control  is  to  provide 
understanding  into  the  project’s  progress  so  that  appropriate 
corrective  actions  can  be  taken  when  the  project’s 
performance  deviates  significantly  from  the  plan. 
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Project  Monitoring  and  Control2 

For  Acquisition,  monitoring  and  control  functions  are 
directed  within  the  acquisition  project  early  in  the  process 
as  the  acquisition  planning  is  performed  and  the  strategy  is 
defined.  As  the  acquisition  process  enfolds,  monitoring  and 
control  are  essential  to  ensuring  that  appropriate  resources 
are  being  applied  and  that  the  internal  acquisition  activities 
are  progressing  according  to  plan. 

Once  a  supplier  is  selected  and  an  award  is  made,  the  role 
of  monitoring  and  control  becomes  two  fold,  concerned  with 
both  continuing  to  monitor  and  control  internally  while  also 
monitoring  and  controlling  the  progress  of  the  supplier’s 
execution  under  the  supplier’s  project  plan. 
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Poor  Project  Monitoring  and  Control... 

Symptoms 

•  Lots  of  time  is  spent  in  meetings  trying  to  discover  project 
status  rather  than  reporting  on  it. 

•  Data  needed  for  management  decisions  is  unavailable  when 
needed. 

•  Actions  that  should  have  been  taken  early  aren’t  identified  until 
it’s  too  late 

Why  should  we  care? 

•  If  you  don’t  know  what’s  going  on,  corrective  action  can’t  be 
taken  early  when  it’s  least  expensive. 

•  Lack  of  management  insight/oversight  makes  project  results 
highly  unpredictable,  even  later  in  the  project. 

•  If  your  confidence  in  the  status  you  give  to  your  customer  is 
low,  they  probably  perceive  it. 
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Project  Monitoring  and  Control 

CMMI-AM  Goals  and  Practices 


Specific  Goal  Specific  Practice 


Monitor  Project 
Against  Plan 

Monitor  Project  Planning  Parameters 

Monitor  Commitments 

Monitor  Project  Risks 

Monitor  Data  Management 

Monitor  Stakeholder  Involvement 

Conduct  Progress  Reviews 

Conduct  Milestone  Reviews 

Manage  Corrective 
Action  to  Closure 

Analyze  Issues 

Take  Corrective  Action 

Manage  Corrective  Action 
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Module  2  Agenda 

Introduction  to  the  CMMI  Acquisition  Module 

Project  Management  Process  Areas 

•  Project  Planning 

•  Project  Monitoring  and  Control 

•  Solicitation  and  Contract  Monitoring 

•  Integrated  Project  Management 

•  Risk  Management 
Summary 
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Solicitation  and  Contract  Monitoring 

The  purpose  of  Solicitation  and  Contract  Monitoring  is  to  prepare 
a  solicitation  package  that  identifies  the  needs  of  a  particular 
acquisition,  to  select  a  supplier  who  is  best  capable  of  satisfying 
those  needs,  and  to  establish  the  process  for  monitoring  the 
supplier  for  the  duration  of  the  contract. 

For  Acquisition,  the  solicitation  must  comply  with  the  applicable 
federal,  departmental,  and  service  acquisition  regulations  and 
policies.  The  solicitation  should  address  issues  appropriate  to  the 
product  domain  or  acquisition  environment  (e.g.,  supplier  process 
evaluations,  operational  safety  suitability  and  effectiveness, 
certifications,  architecture  evaluations,  and  interoperability).  The 
representatives  responsible  for  these  activities  within  the  project 
or  stakeholder  organizations  should  be  consulted  for  proper 
inclusion  of  those  activities  into  the  solicitation  and  contract 
monitoring  process. 
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Poor  Solicitation  and  Contract  Monitoring... 

Symptoms 

•  The  solicitation  package  does  not  include  the 
agreement/contractual  requirements  and  proposal  evaluation 
criteria. 

•  The  technical  and  management  elements  of  proposals  are  not 
properly  evaluated  to  ensure  that  the  requirements  of  the 
agreement/contract  will  be  satisfied. 

•  The  selection  official  will  not  select  suppliers  who  are  qualified  to 
satisfy  the  agreement/contract’s  requirements  for  the  project’s 
products.. 

Why  Do  We  Care? 

•  The  project  team  will  have  insufficient  insight  into  the  supplier’s 
activities  to  ensure  the  effort  is  managed,  controlled  and  complies 
with  contract  requirements. 

•  The  project  team  and  supplier  team  will  be  unable  to  maintain 
ongoing  communication  and  commitments. 
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Solicitation  and  Contract  Monitoring 

CMMI-AM  Goals  and  Practices 


Specific  Goal 

Specific  Practice 

Prepare  for  the 
Solicitation 

•  Designate  a  Selection  Official 

•  Establish  a  Solicitation  Package  and  Evaluation 
Criteria 

•  Establish  Cost  and  Schedule  Estimates 

•  Validate  the  Solicitation  Package 

Select  Suppliers 

•  Evaluate  Proposals 

•  Use  Evaluation  Results  to  Select  Suppliers 

Award  Contracts 

•  Establish  an  Understanding  of  the  Contract  and 
Proposed  Approach 

•  Establish  Communications  Processes  and 
Procedures 

Coordinate  Work 
with  Suppliers 

•  Monitor  Selected  Supplier  Processes 

•  Evaluate  Selected  Supplier  Work  Products 

•  Revise  the  Supplier  Agreement  or  Relationship 
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Solicitation  and  Contract  Monitoring 

Goal  1:  Prepare  for  the  Solicitation  i 

The  project  is  prepared  to  conduct  the  solicitation 

Designate  a  selection  official  responsible  for  making  the 
selection  decision 

Establish  and  maintain  a  solicitation  package  that  includes 
the  needs  of  the  acquisition  and  corresponding  proposal 
evaluation  criteria 

•  Define  the  required  proposal  content 

-  Process  descriptions  and  commitments 

-  Proposed  development  approach  (e.g.,  processes,  tasks, 
activities) 

-  Metrics  to  be  provided  to  the  PMO  (including  process 
metrics) 

-  Appropriate  plans  (e.g.,  Integrated  Mg’t  plan,  Software 
Development  Plan,  risk  Management  Plan) 
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Goal  1:  Prepare  for  the  Solicitation  2 

Establish  and  maintain  independently  reviewed  cost  and 
schedule  estimates  for  the  products  to  be  acquired 

•  Reviewers  should  not  be  connected  with  the  acquisition  team 
or  the  supplier 

Validate  the  solicitation  package  with  end  users  and 
potential  offerors  to  ensure  the  approach  and  cost  and 
schedule  estimates  are  realistic  and  can  reasonably  lead  to  a 
usable  product. 

•  In  a  competitive  environment,  ensure  equal  access  to  all 
potential  offerors.  Provide  a  means  for  reviewers  to  offer 
clarifications  of  ambiguous  capabilities. 

•  In  a  sole  source  or  change  order  environment,  ensure  that 
relevant  stakeholders  recognize  the  consequences  of 
proposed  changes 
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Goal  2:  Select  Supplier 

Suppliers  are  selected  based  on  the  solicitation 
package 

Evaluate  proposals  according  to  the  documented  evaluation 
criteria 

•  In  addition  to  evaluating  the  technical  approach,  evaluate 

-  Management  practices  -  Sufficiency  of  plans 

-  Process  capabilities  -  Domain  experience 

-  Cost  -  Schedule 

-  Past  Performance 

Use  proposal  evaluation  results  as  a  basis  to  support 
selection  decisions 
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Goal  3:  Award  Contracts  i 


Contracts  are  issued  based  on  the  needs  of  the 
acquisition  and  the  suppliers’  proposed  approaches 

Establish  and  maintain  a  mutual  understanding  of  the 
contract  with  selected  suppliers  and  end  users  based  on  the 
acquisition  needs  and  the  suppliers’  proposed  approaches 

•  Ensure  that  contractual  commitments  are  made  for  factors 
critical  to  project  success  (e.g.,  process  execution,  metrics 
collection  and  reporting) 

•  Maintain  mutual  understanding  for  the  duration  of  the  contract 
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Goal  3:  Award  Contracts  2 

Establish  and  maintain  communication  processes  and 
procedures  with  suppliers  that  emphasize  the  needs, 
expectations,  and  measures  of  effectiveness  to  be  used 
throughout  the  acquisition 

•  Define  ground  rules  for 

-  communication  (e.g.,  data  reported,  frequency  of  reporting) 

-  key  decision-making  (e.g.,  rationale,  documentation, 
acquirer  involvement) 

-  conflict  resolution 

•  Monitor  process  deployment  and  effectiveness 

•  Maintain  open  lines  of  communication 
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Goal  4:  Coordinate  Work  i 


Work  is  coordinated  with  suppliers  to  ensure  the 
contract  is  executed  properly 

Monitor  and  evaluate  selected  processes  used  by  the 
supplier  based  on  the  supplier’s  documented  processes 

•  Adherence  to  plan 

•  Timeliness  of  deployment 

•  Effectiveness  of  process 

Evaluate  selected  supplier  work  products  based  on 
documented  evaluation  criteria 

•  Define  work  products  to  be  evaluated  (may  include  interime 
products)  and  evaluation  criteria 

•  Ensure  capacity  and  capability  for  timely  and  accurate 
evaluation 
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Goal  4:  Coordinate  Work  2 

Revise  the  supplier  agreement  or  relationship,  as 
appropriate,  to  reflect  changes  in  conditions 

•  Address  shortfalls  in  both  products  and  processes 

•  Offer  relief  when  needs  evolve  to  invalidate  process 
requirements,  documentation  requirements,  reporting 
requirements,  etc. 
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Module  2  Agenda 

Introduction  to  the  CMMI  Acquisition  Module 

Project  Management  Process  Areas 

•  Project  Planning 

•  Project  Monitoring  and  Control 

•  Solicitation  and  Contract  Monitoring 

•  Integrated  Project  Management 

•  Risk  Management 
Summary 
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Integrated  Project  Management 

For  Acquisition,  integrated  project  management  involves 
establishing  project  management  processes  consistent  with 
and  tailored  from  the  organizations  standard  processes.  This 
includes  higher  level  acquisition  guidance,  regulations, 
instructions,  as  well  as  local  practices  established  to  be  used 
across  various  projects  in  the  local  organization.  Establishing 
an  integrated  project  management  process  incorporating  and 
involving  all  stakeholders  (executive  level  acquisition  offices, 
users,  test  organizations,  developers,  and  associated 
government  support  organizations)  is  critical  to  the 
successful  development  of  the  project. 

Formal  interfaces  among  project  stakeholders  take  the  form 
of  memorandums  of  understanding  (MOUs),  memorandums 
of  agreements  (MOAs),  contractual  commitments,  associate 
contractor  agreements  and  similar  documents  depending  on 
the  nature  of  the  interfaces  and  involved  stakeholders. 
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Poor  Integrated  Project  Mg’t ... 

Symptoms 

•  No  defined  processes  for  the  project 

•  Project  estimates  make  no  reference  to  prior  projects 

•  Plans  do  not  reflect  the  way  the  project  is  executed 

•  Project  staff  does  not  know  what  is  in  the  project  plans 

•  Stakeholders  are  not  identified  and  involved 

Why  do  we  care? 

•  Without  processes,  performance  is  ad  hoc 

•  Without  the  history  of  prior  projects,  we  may  make  the  same 
mistakes 

•  If  execution  doesn’t  follow  the  plans,  what  does  it  follow? 

•  Uninvolved  stakeholders  can  provide  last-minute  surprises 

•  Lessons  learned  are  not  captured 
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CMMI-AM  Goals  and  Practices 


Specific  Goal 

Specific  Practice 

Use  the  Project’s 
Defined  Process 

•  Establish  the  Project’s  Defined  Process 

•  Use  Organizational  Process  Assets  for  Planning 
Project  Activities 

•  Integrate  Plans 

•  Manage  the  Project  Using  the  Integrated  Plans 

•  Contribute  to  the  Organizational 

Process  Assets 

Coordinate  and 
Collaborate  with 
Relevant 
Stakeholders 

•  Manage  Stakeholder  Involvement 

•  Manage  Dependencies 

•  Resolve  Coordination  Issues 
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Module  2  Agenda 

Introduction  to  the  CMMI  Acquisition  Module 

Project  Management  Process  Areas 

•  Project  Planning 

•  Project  Monitoring  and  Control 

•  Solicitation  and  Contract  Monitoring 

•  Integrated  Project  Management 

•  Risk  Management 
Summary 
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Risk  Management 

The  purpose  of  risk  management  is  to  identify  potential  problems 
before  they  occur,  so  that  risk-handling  activities  may  be  planned 
and  invoked  as  needed  across  the  life  of  the  product  or  project  to 
mitigate  adverse  impacts  on  achieving  objectives. 

For  Acquisition,  risk  identification  and  estimation  of  probability  of 
occurrence  and  impact,  particularly  for  those  risks  involved  in 
meeting  performance  requirements,  schedules,  and  cost  targets, 
largely  determines  the  acquisition  strategy.  The  acquirer  has  a  dual 
role,  first  in  assessing  and  managing  overall  project  risks  for  the 
duration  of  the  project,  and  second,  in  assessing  and  managing  risks 
associated  with  the  performance  of  the  supplier.  As  the  acquisition 
progresses  to  the  selection  of  a  supplier,  the  risk  specific  to  the 
supplier’s  technical  and  management  approach  becomes  important 
to  the  success  of  the  acquisition. 


©  2005  by  Carnegie  Mellon  University 


Tutorial:  Use  of  CMMI®  in  Acquisition  Environments  -  Page  57 


Carnegie  Mel  Ion 

Software  Engineering  Institute 

Poor  Risk  Management ... 

Symptoms 

•  Risks  are  being  ignored. 

•  Known  risks  to  project  staff  are  a  surprise  to  management. 

•  Every  time  a  new  problem  manifests,  a  new  management 
technique  is  tried. 

Why  should  we  care? 

•  The  project  may  escape  some  of  the  “bullets,”  but  not  all  of 
them. 

•  No  lessons  learned  for  future  projects  means  making  the  same 
mistakes  on  multiple  projects. 

•  Repeated  project  failures  due  to  unforeseen  (but  predictable) 
risks  costs  you  and  your  organization. 
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CMMI-AM  Goals  and  Practices 


Specific  Goal 

Specific  Practice 

Prepare  for  Risk 
Management 

•  Determine  Risk  Sources  and  Categories 

•  Define  Risk  Parameters 

•  Establish  a  Risk  Management  Strategy 

Identify  and 
Analyze  Risks 

•  Identify  Risks 

•  Evaluate,  Categorize,  and  Prioritize  Risks 

Mitigate  Risks 

•  Develop  Risk  Mitigation  Plans 

•  Implement  Risk  Mitigation  Plans 
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Acquisition  Risk  Management  i 

What  Can  Acquisition  Program  Offices  Do?  —  A  Few  Ideas 

Start  a  risk  management  program  on  Day  1  of  the  program 

Ensure  that  PMO  staff  have  appropriate  risk  management 
training 

Use  multiple  methods  to  identify  risk  sources: 

-  periodic  risk  reporting  -  brainstorming 

-  voluntary  risk  reporting  -  risk  report  forms 

-  taxonomy-based  questionnaire  (TBQ)  -  TBQ  interviews 
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What  Can  Acquisition  Program  Offices  Do?  —  A  Few  Ideas 

•  Add  language  to  RFPs  and  contracts  that  specify  how  risks  are 
to  be  reported  to  the  PMO 

•  Encourage  decentralization  of  risk  identification  and  analysis 
following  an  organizationally  defined  process 

•  Establish  and  maintain  a  schedule  of  joint  risk  reviews  with  all 
contractors  throughout  the  program,  including  joint  prioritization 
of  the  most  important  risks  to  the  program 

•  Find  ways  to  reward  contractors  for  early  identification  of 
issues  and  risks 

•  Define  a  process  and  criteria  for  escalating  risks  to  the  next 
higher  level 
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Summary  i 

PM  roles  include  PMO  management,  project  management, 
supplier  oversight,  indirect  subcontractor  oversight,  and  process 
integration 

CMMI-AM  is  a  tool  intended  to  help  the  acquirer  achieve  success 

Development  of  a  suitable  acquisition  strategy  is  a  key 
component  of  project  planning 

Principal  goals  of  Project  Planning 

•  Establish  estimates 

•  Develop  a  project  plan 

•  Obtain  commitment  to  the  plan 

Principal  goals  of  Project  Monitoring  and  Control 

•  Monitor  Project  Against  Plan 

•  Manage  Corrective  Action  to  Closure 
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Principal  goals  of  Solicitation  and  Contract  Monitoring 

•  Prepare  for  the  Solicitation 

•  Select  Suppliers 

•  Award  Contracts 

•  Coordinate  Work  with  Suppliers 

Principal  goals  of  Integrated  Project  Management 

•  Use  the  Project’s  Defined  Process 

•  Coordinate  and  Collaborate  with  Relevant  Stakeholders 

Principal  goals  of  Risk  Management 

•  Prepare  for  Risk  Management 

•  Identify  and  Analyze  Risks 

•  Mitigate  Risks 
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Summary 
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PMO  Role  in  Systems  Engineering  i 


CMMI 


® 


Inherent  PMO  Responsibility: 

•  Ensure  technology  readiness  level  is  appropriate  for  program 
phase 

•  Develop  initial  system  requirements  in  conjunction  with 
stakeholders  and  ensure  continued  involvement 

•  Develop  technical  evaluation  criteria  and  evaluate  proposals 
during  source  selection 

•  Develop  independent  cost  and  schedule  estimates  for  the 
technical  effort 

•  Ensure  external  interfaces  are  properly  identified  and 
monitored 

•  Ensure  PMO  has  adequate  systems  engineering  staff 
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PMO  Role  in  Systems  Engineering  2 


CMMI 


® 


PMO  responsibility  in  conjunction  with  your  contractor: 

•  Ensure  contractor  development  method  is  appropriate 

•  Ensure  contractor’s  systems  engineering  processes  are 
acceptable  and  being  followed 

•  Ensure  compatible  processes  between  prime  and  sub 
contractors  and  between  the  contractor  team  and  the  PMO 

•  Review  and  approve  systems  engineering  documentation 

•  Ensure  systems  engineering  function  is  adequately  integrated 
with  other  areas  such  as  logistics  and  test 

•  Manage  the  top-level  change  control  process 

•  Perform  technical  evaluations 

•  Systems  Integration  (if  applicable) 

•  Ensure  end  system  meets  requirements 
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CMMI 
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Engineering  Process  Areas 

•  Requirements  Development 

•  Requirements  Management 

•  Verification 

•  Validation 
Summary 
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Requirements  Development  i 


The  purpose  of  requirements  development  is  to  produce 
and  analyze  customer,  product,  and  product-component 
requirements. 
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Requirements  Development  2 

The  purpose  of  requirements  development  is  to  produce 
and  analyze  customer,  product,  and  product-component 
requirements. 

For  Acquisition,  requirements  development  has  two 
contexts; 

•  The  amalgamation  and  coordination  of  the  operational 
requirements  (customer  requirements)  into  a  requirements 
set  that  will  define  the  scope  and  direction  of  the  acquisition; 

•  The  allocation  and  extension  of  the  customer  requirements 
and  additional  acquirer  requirements  (e.g.,  architecture, 
formal  and  informal  reviews,  reporting  or  data  requirements) 
that  become  the  basis  of  the  processes  utilized  by  the 
supplier’s  organization. 
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Requirements  Development  3 


There  is  a  continuous  iteration  of  requirements  down  through 
the  multiple  tiers  of  requirements  documents  associated  with  the 
components  of  the  system. 

•  For  example,  requirements  flow  from  the  stakeholders  to  the 
system  level  to  multiple  subsystem  levels  and  eventually  to 
either  hardware  or  software  component  levels. 

The  responsibility  for  developing  requirements  across  the  levels 
is  generally  split  between  the  acquirer  and  the  supplier. 

•  The  acquirer  is  generally  responsible  for  the  higher  level, 
starting  with  operational  requirements  and  the  supplier  is 
responsible  for  successive  levels  below  that. 
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Poor  Requirements  Development ... 


Symptoms 

•  Unstated  requirements  or  poorly  stated  requirements  lead  to 
confusion  among  staff  and  customers. 

•  Design,  implementation,  and  test  work  products  inconsistently 
interpret  the  requirements. 

•  It  takes  a  long  time  to  get  agreement  on  product  design. 

Why  should  we  care? 

•  Unusable  products  and  unhappy  customers 

•  Wasted  time  and  resources  building  the  “wrong”  product 

•  Staff  members  get  tired  of  rework  because  requirements  have 
been  re-interpreted  yet  again. 

•  Excessive  spending  to  satisfy  customer  expectations 
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Requirements:  Input  or  Output? 


You  RECEIVE  requirements  from  your  customer 

•  Operational  needs 

You  DELIVER  requirements  to  your  supplier 

•  Through  the  solicitation,  SOW,  SOO,  and/or  contract 

Your  job  is 

•  To  ensure  the  quality  of  the  inputs 

•  To  convert  the  inputs  to  the  high-quality  outputs 
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Requirements  Must  be  Balanced 
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CMMI-AM  Goals  and  Practices 


Specific  Goal  Specific  Practice 


Develop 

Customer 

Requirements 

Elicit  Needs 

Develop  the  Customer  Requirements 

Develop 

Product 

Requirements 

Establish  Product  and  Product-  Component 
Requirements 

Allocate  Product-Component  Requirements 

Identify  Interface  Requirements 

Analyze  and 

Validate 

Requirements 

Establish  Operational  Concepts  and  Scenarios 
Establish  a  Definition  of  Required  Functionality 
Analyze  Requirements 

Analyze  Requirements  to  Achieve  Balance 

Validate  Requirements  with  Comprehensive 

Methods 
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Engineering  Process  Areas 

•  Requirements  Development 

•  Requirements  Management 

•  Verification 

•  Validation 
Summary 
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Requirements  Management  i 

The  purpose  of  requirements  management  is  to 
manage  the  requirements  of  the  project’s  products  and 
product  components  and  to  identify  inconsistencies 
between  those  requirements  and  the  project’s  plans  and 
work  products. 

For  Acquisition,  requirements  management  is  applied  to 
the  requirements  that  are  received  from  the 
requirements  development  process. 
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Requirements  Management  2 

During  acquisition,  requirements  management  includes 

•  the  direct  management  of  acquirer-controlled 
requirements 

•  oversight  of  supplier  requirements  management 

Requirements  are  managed  and  maintained  with 
discipline  so  that  changes  are  not  executed  without 
recognizing  the  impact  to  the  project. 

Requirements  management  does  not  end  with  the 
selection  of  a  supplier  and  an  award. 

•  The  acquisition  project  continues  to  manage  high-level 
requirements,  including  changes 

•  the  selected  supplier  manages  the  lower  level 
requirements 
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Symptoms 

•  High  levels  of  re-work  throughout  the  project. 

•  Requirements  are  accepted  by  staff  from  any  unauthorized  sources. 

•  “Galloping”  requirements  creep. 

•  Inability  to  prove  that  the  product  meets  the  approved  requirements 

Why  should  we  care? 

•  Solutions  that  don’t  match  user  needs  or  may  have  to  be  replaced  or 
retired  early 

•  Inability  to  hold  contractor  to  commitments 

•  Excessive  budget  consumption  [LEFF  2003] 

-  Requirements  errors  are  the  most  common  error  &  most  expensive 
to  fix 

-  Requirements  error  are  likely  to  consume  25%  -  40%  of  the  total 
project  budget  when  not  caught  early 
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Requirements  Management 

CMMI-AM  Goals  and  Practices  i 


Specific  Goal  Specific  Practice 


Manage 

Requirements 


Obtain  an  Understanding  of  Requirements 

Obtain  Commitment  to  Requirements 

Manage  Requirement  Changes 

Maintain  Bidirectional  Traceability  of 
Requirements 

Identify  Inconsistencies  Between 
Project  Work  and  Requirements 
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Module  3  Agenda 

Engineering  Process  Areas 

•  Requirements  Development 

•  Requirements  Management 

•  Verification 

•  Validation 
Summary 
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Verification 

•  Are  you  building  the  product  right ? 

•  That  is,  are  you  meeting  the  specified  requirements? 


Validation 

•  Are  you  building  the  right  product ? 

•  That  is,  are  you  meeting  the  operational  need? 
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Verification 

The  purpose  of  verification  is  to  ensure  that  selected  work 
products  meet  their  specified  requirements. 

For  Acquisition,  verification  involves  ensuring  that  the  evolving 
work  products  of  the  acquisition  project  meet  specified 
requirements  for  those  products.  The  acquisition  project  should 
ensure 

•  a  proper  verification  environment  exists 

•  work  products  are  selected  for  evaluation  based  on 
documented  criteria. 

Peer  reviews  are  intended  to  be  used  for  work  products 
developed  by  the  acquisition  project 

The  acquisition  project  is  also  responsible  for  ensuring  that  the 
supplier  uses  appropriate  methods  to  verify  its  work  products. 
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Poor  Verification  ... 


CMMI 
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Symptoms 

•  There  is  disagreement  among  technical  staff  as  to  the  “done¬ 
ness”  of  different  components. 

•  Under  test  the  product  doesn’t  meet  requirements  or  design 
expectations. 

•  Defects  that  could  have  been  caught  early  escape  into  later  life 
cycle  phases. 

•  There  is  increased  integration  or  test  time. 

Why  should  we  care? 

•  Product  reliability  suffers  if  defects  aren’t  detected  or  corrected 
prior  to  customer  release. 

•  The  product  costs  more  to  test  if  early  verification  activities  are 
ignored. 

•  Customers  don’t  want  to  pay  for  defective  products,  and  you 
probably  won’t  get  their  business  next  time 
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Module  3  Agenda 

Engineering  Process  Areas 

•  Requirements  Development 

•  Requirements  Management 

•  Verification 

•  Validation 
Summary 
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Validation  i 
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The  purpose  of  Validation  is  to  demonstrate  that  a  product  or 
product  component  fulfills  its  intended  use  when  placed  in  its 
intended  environment. 

Validation  activities  can  be  applied  to  all  aspects  of  the  product  in 
any  of  its  intended  environments 

•  e.g.,  operation,  training,  manufacturing,  maintenance,  and 
support  services. 

The  methods  employed  to  accomplish  validation  can  be  applied 

to  work  products  as  well  as  to  the  product  and  product 
components. 

The  work  products  (e.g.,  requirements,  designs,  prototypes) 
should  be  selected  for  validation  based  on  which  are  the  best 
predictors  of  how  well  the  delivered  end  product  and  product 
components  will  satisfy  user  needs. 
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For  acquisition,  validation  involves  ensuring  that  the  evolving 
acquisition  work  products  (e.g.,  RFPs,  SOWs,  plans)  meet  the 
acquisition  project’s  needs. 

Validation  activities  are  normally  performed  early  and 
continuously  throughout  the  acquisition  life  cycle. 

The  acquirer  also  uses  validation  processes  to  ensure  that  the 
product  or  service  received  from  the  supplier  will  fulfill  its  intended 
use. 

The  test  community  is  a  major  stakeholder,  participating  in  up¬ 
front  planning  through  final-product  acceptance. 

•  The  supplier  and/or  the  test  community  may  perform  many  of 
the  validation  practices,  with  the  acquisition  project  facilitating 
the  correction  of  deficiencies  or  enhancements  by  the  supplier 
or  follow-on  maintenance  organization. 
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Symptoms 

•  Lots  of  user  change  requests  are  received  before  or  soon 
after  the  product  is  released. 

•  There  are  arguments  among  the  technical  staff  as  to 
what  the  user  really  wants. 

•  The  released  product  doesn’t  meet  user  expectations. 

Why  should  we  care? 

•  Customers  don’t  want  to  pay  for  products  that  don’t  meet 
their  needs. 

•  If  an  end  user  refuses  to  use  the  product  as  delivered, 
their  confidence  in  you  is  eroded. 

•  You’ll  spend  a  lot  of  money  trying  to  make  it  right,  or 
you’ll  give  up  that  customer’s  future  business. 
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Validation 

CMMI-AM  Goals  and  Practices 


Specific  Goal  Specific  Practice 


Prepare  for 
Validation 


•  Select  Products  for  Validation 

•  Establish  the  Validation  Environment 


•  Establish  Validation  Procedures 
and  Criteria 


Validate 
Product  of 
Product 
Components 


Perform  Validation 
Analyze  Validation  Results 


©  2005  by  Carnegie  Mellon  University 


CMMI-AM  and  Engineering  vO.1 


CMMI  Acquisition  Module  -  Page  M3-28 


Carnegie  Mellon 

Software  Engineering  Institute 


Module  3  Agenda 

Engineering  Process  Areas 

•  Requirements  Development 

•  Requirements  Management 

•  Verification 

•  Validation 
Summary 
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PMO  plays  a  critical  role  in  the  systems  engineering  of  a  project 

Principal  goals  of  Requirements  Development 

•  Develop  Customer  Requirements 

•  Develop  Product  Requirements 

•  Analyze  and  Validate  Requirements 

Principal  goals  of  Requirements  Management 

•  Manage  Requirements 
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Summary  2 

Principal  goals  of  Verification 

•  Prepare  for  Verification 

•  Perform  Peer  Reviews 

•  Verify  Selected  Work  Products 

Principal  goals  of  Validation 

•  Prepare  for  Validation 

•  Validate  Product  of  Product  Components 
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Using  CMMI-AM  to  Improve  Acquisition 
Practices-  Contents 


Module  1  - 

Background 

Tutorial  information  and  background 

Module  2  - 

CMMI-AM  and  Project  Management 

Project  Management  process  areas,  goals,  and 
practices 

Module  3  - 

CMMI-AM  and  Engineering 

Engineering  process  areas,  goals,  and  practices 

Module  4- 

CMMI-AM  and  Support  and  Generic  Practices 

Support  process  areas,  goals,  and  practices;  and 

Generic  Practices 

Module  5  - 

Using  CMMI-AM 

Module  6  - 

Summary  and  Conclusion 
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Using  CMMI-AM  to  Improve 
Acquisition  Practices 


Module  4: 

CMMI-AM  and 
Support  and  Generic 
Practices 
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Support  Process  Areas 

•  Decision  Analysis  and  Resolution 

•  Measurement  and  Analysis 

•  Transition  to  Operations  and  Support 
Summary 

Generic  Practices 
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Decision  Analysis  and  Resolution 

The  purpose  of  decision  analysis  and  resolution  is  to  analyze 
possible  decisions  using  a  formal  evaluation  process  that 
evaluates  identified  alternatives  against  established  criteria. 

For  Acquisition,  a  repeatable  criteria-based  decision-making 
process  is  especially  important,  both  while  making  the  critical 
decisions  that  define  and  guide  the  acquisition  process  itself  and 
later  when  critical  decisions  are  made  with  the  selected  supplier. 
The  establishment  of  a  formal  process  for  decision-making 
provides  the  acquisition  project  with  documentation  of  the 
decision  rationale.  Such  documentation  allows  the  criteria  for 
critical  decisions  to  be  revisited  when  changes  that  impact 
project  requirements  or  other  critical  project  parameters  change. 
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Poor  Decision  Analysis  and  Resolution  ... 

Symptoms 

•  It  is  unclear  who  is  authorized  to  make  what  decisions. 

•  Decisions  are  made  on  primarily  subjective  bases. 

•  The  same  issue  is  “decided”  over  and  over  and  over. 

•  Rationale  for  earlier  decisions  is  unavailable  when  needed  to 
understand  the  decision  later  in  the  project. 

•  Too  few  choices  are  considered  for  major  decisions. 

Why  should  we  care? 

•  Wasted  effort  pursuing  changing  goals 

•  Lost  opportunities 

•  Low  morale 

•  Perception  of  indecisiveness  (or  incompetence)  by  customer 
and  others 
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Decision  Analysis  and  Resolution 

CMMI-AM  Goals  and  Practices 


Specific  Goal 

Specific  Practice 

Evaluate 

Alternatives 

•  Establish  Guidelines  for  Decision  Analysis 

•  Establish  Evaluation  Criteria 

•  Identify  Alternative  Solutions 

•  Select  Evaluation  Methods 

•  Evaluate  Alternatives 

•  Select  Solutions 

©  2005  by  Carnegie  Mellon  University 


M4  -  CMMI-AM  and  Support  vO.1 


CMMI  Acquisition  Module  -  Page  M4-6 


Carnegie  Mellon 

Software  Engineering  Institute 


Module  4  Agenda 


CMMI 


® 


Support  Process  Areas 

•  Decision  Analysis  and  Resolution 

•  Measurement  and  Analysis 

•  Transition  to  Operations  and  Support 
Summary 

Generic  Practices 
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The  purpose  of  measurement  and  analysis  is  to  develop  and 
sustain  a  measurement  capability  that  is  used  to  support 
management  information  needs. 

For  Acquisition,  the  acquisition  project  has  information  needs  for 
determining  the  status  of  its  activities  throughout  the  lifecycle  of 
the  acquisition,  the  supplier’s  activities  per  contractual 
requirements,  and  the  status  of  the  evolving  products  acquired.  In 
acquisition  projects  where  multiple  products  are  acquired  to 
deliver  a  capability  to  the  end-user,  or  where  there  are  teaming 
relationships  with  other  acquisition  projects  to  acquire  joint 
capabilities,  additional  information  needs  may  be  identified  to 
ensure  programmatic,  technical,  and  operational  interoperability 
product  objectives  are  identified,  measured,  and  achieved. 
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Symptoms 

•  Management  lacks  objective  data  for  decision  making 

•  Decisions  are  based  upon  intuition 

•  Status  of  project  is  not  clearly  known 

•  No  historical  data  is  available  for  reference 
Why  should  we  care? 

•  Bad  data  or  No  data  =^>  Bad  decisions 

•  Issues  remain  undetected  until  they  blossom  into  problems 

•  No  data  =^>  No  learning  =^>  Repeated  mistakes 
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Roles  and  Information  Exchange 


PMO 


Pre-award 

activities 

•  RFP  prep. 

•  Contract 
Award 

Post-award 

activities 

•  monitor  & 
oversee 
progress 

•  quality  of 
tangibles 
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PMO  Major  Responsibilities 

Post  Contract  Award 


Deliverables 


PMO 


Contractor 


Develop 

the 

System 


Documents 

-  SRD 

-  SDP 

-  Meas  Plan 
-SDD 

-  Etc. 

Status  Rpts 

-  Sched. 

-  Cost 

-  Testing 

-  Etc. 

Final 

Product 


PMO  Responsibilities 
(Post  Contract  Award) 


•  Evaluate  Quality  of 
deliverables 

•  Monitor  and  Oversight 

-  Schedule  &  Progress 

-  Resources  &  Costs 

-  Developer’s  Processes 
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Evaluate  Quality  of  Deliverables 


Measurable  Results  (Examples) 

Products 

•  defects  discovered 

-  description,  severity,  class, 
type 

•  size  of  the  work  product 
Process 

•  effort  invested  in  the  inspection 
process 

•  time  spent  during  the  inspection 
activities 

i 


Indicators 
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Monitor  and  Oversee 
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Status 

Information 


•  schedule 
progress 

•  budget  status 

•  test  results 

•  process 
results,  e.g. 
inspections 

•  Process 
compliance 

•  etc. 


PMO’s 
Analysis  & 
Review 
Process 


PMO's 


Evaluation 

Criteria 


Measurable  Results  (Examples) 

•  contractor  effort  actual  vs.  plan 

•  contractor  schedule  actual  vs. 
plan 

•  defects  reported 

•  description,  severity,  class, 
type 

•  size,  complexity  of  the  work 
product 

l 


Indicators 
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PMO  vs.  Contractor  Focus 


PMO 


Key  Management  Issues 


Contractor’s  Performance 

•  Schedule  &  Progress 

•  Resources  &  Cost 

•  Product  Quality 

PMO’s  Performance 

•  Schedule  &  Progress 

•  Resources  &  Cost 

•  Product  Quality 

PMO’s  Processes 

•  Documented 

•  Improvements 


Contractor 


Schedule  &  Progress 
Resources  &  Cost 
Product  Size  &  Stability 
Product  Quality 
Process  Performance 
Technology  Effectiveness 
Customer  Satisfaction 
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Measurement  and  Analysis 

CMMI-AM  Goals  and  Practices 


Specific  Goal 

Specific  Practice 

Align 

Measurement 
and  Analysis 
Activities 

•  Establish  Measurement  Objectives 

•  Specify  Measures 

•  Specify  Data  Collection  and  Storage  Procedures 

•  Specify  Analysis  Procedures 

Provide 

•  Collect  Measurement  Data 

Measurement  •  Analyze  Measurement  Data 


Results 

•  Store  Data  and  Results 

•  Communicate  Results 
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Support  Process  Areas 

•  Decision  Analysis  and  Resolution 

•  Measurement  and  Analysis 

•  Transition  to  Operations  and  Support 
Summary 

Generic  Practices 
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Transition  to  Operations  and  Support  i 


The  purpose  of  transition  to  operations  and  support  is  to 
provide  for  the  transition  of  the  product  to  the  end  user  and 
the  eventual  support  organization  and  to  accommodate 
lifecycle  evolution  of  the  product. 

For  acquisition,  Transition  to  Operations  and  Support 
involves 

•  the  processes  used  to  plan  for  and  manage  the  transition 
of  new  or  evolved  products  into  operational  use 

•  their  transition  to  the  eventual  maintenance  or  support 
organization. 

•  any  special  conditions  that  may  apply  during  the  eventual 
decommissioning  or  disposal  of  the  products. 
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Transition  to  Operations  and  Support  2 


The  acquisition  project  is  responsible  for  ensuring 

•  the  acquired  products  meet  specified  requirements  (see 
Verification) 

•  can  be  used  in  the  intended  environment  (see  the 
Validation) 

•  can  be  transitioned  into  operational  use  to  achieve  the 
users’  desired  mission  capabilities  and  can  be 
maintained  and  sustained  over  their  intended  life  cycles. 
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Transition  to  Operations  and  Support  3 


The  acquisition  project  is  responsible  for 

•  ensuring  reasonable  planning  for  transition  into 
operations  is  conducted 

•  clear  transition  criteria  exist  and  are  agreed  to  by 
relevant  stakeholders 

•  planning  is  completed  for  product  maintenance  and 
support  of  products  after  they  become  operational. 

These  plans  include  reasonable  accommodation  for  known 
and  potential  evolution  of  the  products  and  their  eventual 
removal  from  operational  use. 
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Poor  Transition  to  Operations  and  Support 


Symptoms 

•  Operational  and  support  functions  are  not  involved  during 
development 

•  Support  concerns  not  addressed  during  development 

•  Training  only  addressed  late  in  the  development  process 

Why  should  we  care? 

•  Product  poorly  received  by  Ops  and  Support 

•  Deployment  delayed  due  to  late  Ops  training  or  support 
training 

•  Excessive  support  costs 
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Transition  to  Operations  and  Support 

CMMI-AM  Goals  and  Practices 


Specific  Goal  Specific  Practice 


Prepare  for 
Transition 


Transition 

Products 


•  Establish  a  Transition  Strategy 

•  Establish  Product  Transition  Plans 

•  Establish  Operations  and  Support  Training 
Requirements 

•  Establish  Lifecycle  Resource  Requirements 

•  Identify  Support  Responsibility 

•  Establish  Enhancement  Criteria 

•  Establish  Transition  Criteria 

•  Evaluate  Product  Readiness 

•  Evaluate  Personnel  Readiness 

•  Analyze  Results  and  Take  Action 


©  2005  by  Carnegie  Mellon  University 


M4  -  CMMI-AM  and  Support  vO.1 


CMMI  Acquisition  Module  -  Page  M4-21 


Carnegie  Mellon 

Software  Engineering  Institute 


Transition  to  Operations  and  Support 

Goal  1:  Prepare  for  Transition  i 


Preparation  for  transition  to  operations  and  support  is 
conducted 

Establish  and  maintain  a  strategy  for  transition  to  operations 
and  support 

•  Source  of  support  (organic,  contractor,  etc.) 

•  Level  of  support  (line,  intermediate,  depot,  etc.) 


Establish  and  maintain  plans  for  transitioning  acquired 
products  into  operational  use  and  support 

•  Documented,  available  to,  and  approved  by  relevant 
stakeholders 


Establish  and  maintain  training  requirements  for  operational 
and  support  personnel 

•  Training  objectives  •  Trainee  skills  assessment 

•  Skills  maintenance 
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Transition  to  Operations  and  Support 

Goal  1:  Prepare  for  Transition  2 


Establish  and  maintain  initial  and  life-cycle  resource 
requirements  for  performing  operations  and  support 

•  Initial  spares  •  Future  spares  and  service 

•  Facilities  •  Disposal 

Identify  and  assign  organizational  responsibility  for  support 

•  Identify  and  involve  EARLY  and  THROUGHOUT  product 
development 

Establish  and  maintain  criteria  for  assigning  responsibility 
for  enhancements 

•  Magnitude  and  complexity  of  enhancement 

•  Required  domain  knowledge  and  experience 

•  Required  acquisition  knowledge 

Establish  and  maintain  transition  criteria  for  the  acquired 
products 

•  Assure  criteria  satisfaction  through  verification  and  validation 
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Transition  to  Operations  and  Support 

Goal  2:  Transition  Products 


Transition  decisions  and  actions  are  executed  in  accordance 
with  transition  criteria 

Evaluate  the  readiness  of  the  acquired  products  to  undergo 
transition  to  operations  and  support 

•  e.g.  Readiness  of  product,  documentation,  training, 
maintenance  equipment,  etc. 

•  Evaluated  throughout  acquisition  life  cycle 

Evaluate  the  readiness  of  the  operational  and  support 
personnel  to  assume  responsibility  for  the  acquired  products 

•  Skills,  training,  staffing,  support  equipment  availability,  etc. 

Analyze  the  results  of  all  transition  activities  and  identify 
appropriate  action 

•  Strengths  and  weaknesses 

•  Actions  to  bolster  weaknesses 
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Support  Process  Areas 

•  Decision  Analysis  and  Resolution 

•  Measurement  and  Analysis 

•  Transition  to  Operations  and  Support 
Summary 

Generic  Practices 
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Summary 

PMO  plays  a  critical  role  in  the  systems  engineering  of  a 
project 

Principal  goals  of  Decision  Analysis  and  Resolution 

•  Evaluate  Alternatives 

Principal  goals  of  Measurement  and  Analysis 

•  Align  Measurement  and  Analysis  Activities 

•  Provide  Measurement  Results 

Principal  goals  of  Transition  to  Operations  and  Support 

•  Prepare  for  Transition 

•  Transition  Products 
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Support  Process  Areas 

•  Decision  Analysis  and  Resolution 

•  Measurement  and  Analysis 

•  Transition  to  Operations  and  Support 
Summary 

Generic  Practices 
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Generic  practices  are  activities  that  ensure  that  the 
processes  associated  with  the  process  area  will  be 
effective,  repeatable,  and  lasting. 

Generic  practices  are  applied  to  EVERY  process  area. 


©  2005  by  Carnegie  Mellon  University 


M4  -  CMMI-AM  and  Support  vO.1 


CMMI  Acquisition  Module  -  Page  M4-28 


Carnegie  Mellon 

Software  Engineering  Institute 


Definitions 


CMMI 


® 


Managed  A  performed  process  that 

Process  •  Is  planned  and  executed  in  accordance  with  policy 

•Employs  skilled  people  having  adequate  resources 
to  produce  controlled  outputs 

•Involves  relevant  stakeholders 

•Is  monitored,  controlled,  and  reviewed 

•  Is  evaluated  for  adherence  to  its  process  description 


Defined  A  Managed  Process  that 

Process  •  Is  tailored  from  the  organization’s  set  of  standard 

processes  according  to  the  organization’s  tailoring 
guidelines 

•  Has  a  maintained  process  description 

•  Contributes  work  products,  measures,  and  other 
process-improvement  information  to  the 
organizational  process  assets 
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Practices  focused  on  institutionalizing  a  Managed  Process 

•  Establish  an  Organizational  Policy 

•  Plan  the  Process 

•  Provide  Resources 

•  Assign  Responsibility 

•  Train  People 

•  Manage  Configurations 

•  Identify  and  Involve  Relevant  Stakeholders 

•  Monitor  and  Control  the  Process 

•  Objectively  Evaluate  Adherence 

•  Review  Status  with  Higher  Level  Management 

Practices  focused  on  institutionalizing  a  Defined  Process 

•  Establish  a  Defined  Process 

•  Collect  Improvement  Information 
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Using  CMMI-AM  to  Improve  Acquisition 
Practices-  Contents 


Module  1  - 

Background 

Tutorial  information  and  background 

Module  2  - 

CMMI-AM  and  Project  Management 

Project  Management  process  areas,  goals,  and 
practices 

Module  3  - 

CMMI-AM  and  Engineering 

Engineering  process  areas,  goals,  and  practices 

Module  4- 

CMMI-AM  and  Support  and  Generic  Practices 

Support  process  areas,  goals,  and  practices;  and 

Generic  Practices 

Module  5  - 

Using  CMMI-AM 

Module  6  - 

Summary  and  Conclusion 
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Module  5: 

Using  CMMI-AM 


SM  CMM  Integration,  IDEAL,  and  SCAMPI  are  service  marks  of  Carnegie  Mellon  University. 

®  Capability  Maturity  Model,  Capability  Maturity  Modeling,  CMM,  and  CMMI  are  registered  in  the  U.S.  Patent  and  Trademark  Office 
by  Carnegie  Mellon  University. 

Sponsored  by  the  U.S.  Department  of  Defense 
©  2005  by  Carnegie  Mellon  University 

This  material  is  approved  for  public  release.  Distribution  is  limited  by  the  Software  Engineering  Institute  to 
attendees. 
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Using  CMMI-AM 


Summary 
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To  guide  the  PM  in  assessing  the  acquisition  program, 
the  CMMI-AM  includes  a  series  of  executive  questions 
focused  on: 

•  Acquisition  Strategy 

•  Acquisition  Planning 

•  Cost  Schedule  and  Performance  Baselines 

•  User  Requirements 

•  Product  Engineering 

•  Acquisition  Processes 

•  Risk  Identification  and  Management 

Questions  are  linked  to  the  CMMI-AM  Process  Areas 
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CMMI-AM  Executive  Questions 

Acquisition  Strategy 

Method  of  Acquisition  Strategy  determination? 
Risk  Mitigation  through  Acquisition  Strategy? 
Stakeholder  involvement  in  Acquisition  Strategy? 
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CMMI-AM  Executive  Questions 

Acquisition  Planning 


Relationship  to  Acquisition  Strategy? 

Program  Scope  determination? 

Determination  of  Development  Effort  size? 
Determination  of  resource  needs? 

Determination  of  critical  path? 

Coordination  of  plans  with  relevant  stakeholders? 
Staffing  with  appropriate  skills  and  experience? 
Ensuring  adequate  supplier  resources? 

Ensuring  adequate  supplier  experience  and  capability? 
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Cost,  Schedule,  and  Performance  Baselines 


Means  of  validating  and  integrating  baselines? 
Provisions  for  independent  reviews? 

Inclusion  of  all  life  cycle  costs? 

Plans  to  track  cost,  schedule,  and  performance? 
Baseline  change  management? 

Evaluation  of  change  impact? 
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CMMI-AM  Executive  Questions 

User  Requirements 


Plans  to  manage  user  involvement? 

Means  of  ensuring  understanding  of  user  needs? 

PMO  role  in  requirements  generation? 

Adaptation  strategy  for  evolving  operations  environment? 


©  2005  by  Carnegie  Mellon  University 


M5-Process  Improvement  vO.1 


CMMI  Acquisition  Module  -  Page  M6-8 


Carnegie  Mellon 

Software  Engineering  Institute 


CMMI-AM  Executive  Questions 

Product  Engineering 


Process  to  define,  verify,  and  validate  requirements  and 
architectures? 

Development  status  monitoring? 

Means  of  incorporating  non-developmental  items  (NDI)? 

Satisfaction  of  NDI  goals? 

NDI  interface  definition  and  acceptance? 

Effort  to  test  and  integrate  NDI? 

Supplier  demonstration  of  performance  and  stability  of 
development  environment  and  tools? 
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CMMI-AM  Executive  Questions 

Acquisition  Process 


Existence,  quality,  and  usage  of  acquisition  processes? 

Monitoring,  control,  and  improvement  of  acquisition 
processes? 

Project  adherence  to  acquisition  processes? 
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CMMI-AM  Executive  Questions 

Risk  Identification  and  Management 


Means  of  identifying  program  risk? 

Risks  related  to  acquisition  strategy  and  plans? 

Risks  associated  with  cost  and  schedule? 

Means  of  ensuring  understand  of  cost  risk? 

Risks  related  to  supplier  execution? 

Risks  outside  of  your  control? 

Analysis  (likelihood  and  impact)  of  program  risks? 

Mitigation  effort  monitoring? 

Risk  management  tools? 

Participants  in  risk  assessment? 

Reserves  for  risk  mitigation  and  problem  impact 
Assessment  of  supplier  mechanisms  for  rapid  process  stand-up 
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CMMI-AM  Self  Assessment  Tool  -i 

The  survey  instrument  is  intended  to  be  used  to  get  a  sense  of  the 
degree  to  which  the  Capability  Maturity  Module  Integration  - 
Acquisition  Module  (CMMI-AM)  Goals  are  implemented  within  a 
particular  program  or  project’s  work  culture. 

Acquisition  practices  within  the  module  are  drawn  and  summarized 
from  existing  sources  of  best  practices: 

Software  Acquisition  Capability  Maturity  Model  (SA-CMM) 
Capability  Maturity  Model  Integration  (CMMI) 

FAA  Integrated  Capability  Maturity  Model  (iCMM) 

Section  804 

This  instrument  will  allow  members  of  the  program  to  give 
anonymous  feedback  on  how  well  they  think  things  are  going,  and 
then  this  information  can  be  conveniently  aggregated  and  presented 
to  program  members  for  discussion  and  problem-solving. 
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CMMI-AM  Self  Assessment  Tool  -2 


Example  Questions 


1  .Estimates  are  based  on  wild  guesses  or  Estimates  of  project  planning 

dictated  from  above.  parameters  (i.e,  scope,  task  attributes, 

lifecycle,  cost,  effort,  etc.)  are 


1 


established  and  main 
7  8  9  10 


ained. 


2. Plans  are  rarely  written  down  nor  do 
they  reflect  current  project  activities. 


A  project  plan  is  established  and 
maintained  as  the  basis  for  managing 

the  project. 

6  7  8  9  10 


3.We  rarely  seek  commitments  from  those 
affected  by  the  project  plan. 

1  2  3  4  5 


Commitments  to  the  project  plan  are 
established  and  maintained. 
:  7  8  9  10 
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CMMI-AM  Self  Assessment  Tool 

Sample  Output 


-3 


CMMI-AM  Goal  Implementation  Survey  -  Project  Management 
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CMMI-AM  Self  Assessment  Tool  -4 
Interpretation 


The  CMMI-AM  Goal  Implementation  Score  indicates  your 
perception  of  overall  level  of  your  program's  current 
acquisition  management  effort  based  on  implementation  of 
the  CMMI-AM. 

High  scores  are  an  indication  that  you  feel  these  goals  are 
being  achieved  within  your  program  and  may  even  be 
institutionalized. 

Scores  in  other  ranges  mean  that  you  must  build  strategies 
to  improve  the  project's  ability  to  effectively  manage  the 
project. 
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Using  IDEAL  to  adopt  CMMI-AM 

IDEAL:  '  ‘  Learnin9 

Initiate,  Diagnose 
Establish,  Act, 

Learning 


1 


Acting 


Stimulus  \ 
for  Change  / 

Set 

'  Context 

Build 

Sponsorship  , 

/  Charter  / 

/  Infra-  / 
f  structure/ 

Initiating  | 

Charact-  l 

I  erize  \ 

Diagnosing 


Establishing 


SM  IDEAL  is  a  service  mark  of  Carnegie  Mellon  University. 
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Using  IDEAL  to  adopt  CMMI-AM  2 


Stimulus  \  Set 
for  Change/  Context 


Something  prompts  a  need  to  improve 
acquisition  practices 

•  Reaction  to  unanticipated  events  or 
circumstances 

•  Edict  from  above 

•  Recognition  that  process  improvement  is  the 
route  to  program  success 


Setting  context  -  make  sure  there  is  consensus  on 

•  The  organization’s  core  mission 

•  Business  goals  and  strategies 

•  A  coherent  vision  for  the  future 

•  A  strategy  to  achieve  that  vision 

•  Models  to  be  used 
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Using  IDEAL  to  adopt  CMMI-AM 


Charter 
Build  /  |nfra_ 

Sponsorship  /  structure, 


Obtain  senior  level  (PEO?)  sponsorship  to 

•  Provide  personal  commitment  to  project 

•  Provide  needed  resources 

•  Change  their  behavior,  if  necessary 

•  Provide  appropriate  rewards 


May  need  to  establish 

•  An  oversight  group 

•  A  change  management  group 

•  One  or  more  Technical  working  groups 
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Using  IDEAL  to  adopt  CMMI-AM  4 


Understand  your  current-state 

•  Start  with  the  CMMI-AM  questionnaire 

•  Consider  an  external  assessment  of  your 
PMO 

•  Learn  more  about  process  improvement 
Define  your  end-state 

•  Establish  goals  for  process  improvement 


•  Establish  a  time  table 

Develop  recommendations 

•  Gap  analysis 

•  Define  improvement  projects 

•  Develop  estimates  for  cost,  schedule,  and  resources 
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Using  IDEAL  to  adopt  CMMI-AM 


Set  priorities 

•  Based  on  urgency  of  need 

•  Based  on  ROI 

•  A  quick  return  bolsters  support  for 
process  improvement 


5 


Plan  Actions 

•  Define  deliverables,  activities  resources 

•  Identify  decision  points 

•  Identify  risks  and  mitigations 

•  Define  schedule  and  milestone 

•  Plan  for  monitoring  and  tracking 
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Using  IDEAL  to  adopt  CMMI-AM  e 


Create  solution 

•  Identify  performance  objectives 

•  Finalize  plans  for  test/pilot  groupt 

•  Construct  the  solution 

Test  the  solution 

•  Train  the  pilot  group 

•  Execute  pilot 

•  Provide  feedback 


Refine  the  solution 

•  Almost  never  works  right  the  first  time  (keep  you  pilots 
small) 

•  Learn  and  repeat 
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Using  IDEAL  to  adopt  CMMI-AM 


Implement  solution 

•  Monitor  during  implementation, 
and  adjust  as  needed 

Analyze  and  validate 

•  What  went  right?  What  went 
wrong? 

•  Were  objectives  met? 


Propose  future  actions 

•  Process  improvement  is  never  complete 

•  Past  success  enables  more  ambitious  future 
projects 
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Summary 

Process  models  provide  a  structured  approach  to  process 
improvement 

Process  improvement  demands  patience,  persistence,  and 
management  support 

Assess  your  program  using  the  questions  of  the  CMMI-AM 

Use  IDEAL  to  implement  CMMI-AM  and  process 
improvement  within  your  PMO 
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Module  6: 

Conclusion 


SM  CMM  Integration,  IDEAL,  and  SCAMPI  are  service  marks  of  Carnegie  Mellon  University. 
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Explicit  Acquisition  Practices 


Operational 

Need 


CMMI  Acquisition  Module 
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